Experimental Evaluation of Feature Selection Methods for Clustering

Background. Recently several clustering methods have been proposed which perform feature selection with the goal of finding structure in high dimensional data that is confined to a small set of features. Most of these methods are stated in terms of non-convex optimization objectives, and are computationally intractable to solve exactly in general. In this project we consider two such methods, both of which are stated as modifications of the famous K–means objective. Approximation algorithms proposed for these methods require initialization with some clustering, which is taken to be the result of standard K–means. Aim. Our goal is to experimentally characterize the behavior of these algorithms, specifically the type of structure that they can and cannot discover, how they differ in terms of performance, and what effect the approximation methods have on the clustering and feature selection results. Data. We use a dataset containing a detailed phenotypic characterization of 378 participants in a lung disease study in terms of 112 demographic, environmental, and medical features. The variable types are mixed, including continuous, ordinal, and nominal features. Methods. Synthetic datasets are designed to elucidate the performance and behavior of the methods in consideration, both in terms of the optimal solutions for the optimization problems and the quality of the solutions obtained from the approximation algorithms. We formulate greedy K–means, a new feature selection method for clustering, inspired by the two methods in consideration. We propose a few initialization methods as alternatives to K–means initialization. We also propose a clustering criterion for mixed variable types. This criterion can directly be adapted to perform feature selection with the same approach used in greedy K–means. Results. Based on synthetic experiments our proposed greedy K–means method seems to never perform much worse than the other two, and it may be advantageous for interpretability because it gives exactly sparse solutions. We show examples where the previously proposed initialization method of using the K–means solution performs arbitrarily poorly, and that alternate initialization is needed. Among the initialization methods we consider, the one that seems to perform best is random support based initialization, and greedy K–means lends itself to that method naturally. We show that the result of applying these methods to the lung disease data is unstable in terms of the resulting clustering, and that the selected features are highly influenced by correlations between variables, which warrants further investigation to ensure there is any significant non-linear structure in the data found by the clustering methods. Conclusions. We provide insights regarding the challenges presented by noisy, high-dimensional data when applying the feature-sparse clustering methods in consideration. We demonstrate the practical importance of not ignoring the approximate nature of the optimization procedures available for these methods. Since linear structure (i.e. correlations) are overwhelmingly likely to exist in real-world data sets, our results indicate that care needs to be taken regarding the type of structure discovered when interpreting the meaning of selected features.